PCI Express (registered trademark) (hereinafter, abbreviated as PCIe (registered trademark)) is used for connecting peripheral equipment or a peripheral device (hereinafter, referred to as an I/O device) and a central processing unit (CPU) in a computer such as a server or a personal computer (PC).
FIG. 18 illustrates a configuration example of a computer 1A employing PCIe. Referring to FIG. 18, the computer 1A includes a CPU 2, a memory controller 3, a memory 4, a PCIe route complex 5A, a PCIe switch 6, and I/O devices 7. Note that the memory controller 3 and the PCIe route complex 5A may be incorporated in the CPU 2.
PCIe configures a tree port type network in which the PCIe route complex 5A is used as a route (hereinafter, this network is referred to as a PCIe fabric). The PCIe switch 6 for switching a PCIe packet, and the I/O devices 7 are connected to the PCIe fabric. Note that the PCIe route complex 5A, the PCIe switch 6, or the I/O device 7 may also be referred to as a PCIe device.
In PCIe, the PCIe route complex 5A or the PCIe switch 6 switches a PCIe packet toward a PCIe device being a destination (e.g. an I/O device 7) by using information on a transaction layer located at a third layer of a PCIe protocol layer.
There are various species of I/O devices 7. For example, there are a network interface card (NIC) for performing communication between computers via a network, a graphics processing unit (GPU) for displaying an image on a display, and a recommended standard 232 version C (RS-232C) device being an interface for serial communication.
These I/O devices 7 transmit and receive data to and from a PCIe fabric at various data transfer rates. For example, in a case of an NIC associated with Ethernet (registered trademark), communication is performed between computers 1A at a communication speed such as 1 Gbps, 2.5 Gbps, 10 Gbps, or 40 Gbps. Therefore, an NIC transmits and receives data to and from the memory 4 at a substantially same data transfer rate in a PCIe fabric via the PCIe route complex 5A and the memory controller 3. This data transmission and reception is referred to as a direct memory access (DMA), and is performed between the memory 4 and an I/O device 7 without using a resource of the CPU 2.
There are two species of DMAs, namely, a DMA Read for reading data from the memory 4, and a DMA Write for writing data in the memory 4. When a DMA Read is performed, in PCIe, first of all, an I/O device 7 issues a memory read request. The memory read request includes information on a length of read data indicating from which area in the memory 4 and how many bytes of data are read.
A memory read request is sent to the memory controller 3 through the PCIe route complex 5A. Further, the memory controller 3 reads, from the memory 4, data in an area designated by the memory read request by a designated size. The read data are transferred to an I/O device 7 that issues the memory read request, through the PCIe route complex 5A. In the PCIe route complex 5A, a header of a packet of a classification called a completion among PCIe packets is attached to the data read from the memory 4, and the data are transmitted to the PCIe fabric.
When a size of data to be read from the memory 4 by using a DMA Read exceeds a size of data transmittable by one PCIe packet, the data read from the memory 4 are divided into a plurality of completion packets, and delivered to an I/O device 7 that issues the memory read request.
In other words, in a DMA Read, whereas a PCIe packet of one memory read request flows in a direction from the I/O device 7 to the PCIe route complex 5A (hereinafter, referred to as an Up direction), one or more PCIe packets flow in a direction from the PCIe route complex 5A to the I/O device 7 (hereinafter, referred to as a Down direction).
In a case of a DMA Write, an I/O device 7 issues a PCIe packet of a memory write request. The memory write request includes an address designating an area being a writing destination, a transfer size, and data to be written.
When a size of data to be written in the memory 4 exceeds a size of data transmittable by one PCIe packet, the data are delivered to the memory 4 by using a plurality of PCIe packets of memory write requests.
Note that a memory write request is a request of a classification called a posted request in PCIe, and a party that receives the request does not have to return a response. This response is a completion pointed out in description about a DMA Read. A memory read request to be issued when a DMA Read is performed is a classification called a non-posted request. In a non-posted request, a party that receives the request always has to return a response (completion).
In other words, in a DMA Write, PCIe packets of a number required by an I/O device 7 flow in an Up direction, and a PCIe packet does not flow in a Down direction.
PTL 1 discloses an I/O controller capable of analyzing a header of a memory read request, determining a priority of processing of the memory read request, based on a transfer size and the like, and inputting the memory read request to a request queue depending on the priority.
The I/O controller in PTL 1 employs a server virtualization environment as a target, and monitors a processing status of a DMA for each virtual machine. Further, when a request accompanying a DMA such as a memory read request or a memory write request is issued from an I/O device, the I/O controller identifies a virtual machine being a transfer destination, and inputs the request to a request queue to which a priority is given, depending on an execution status of a DMA of the virtual machine.
In this way, there is a technique of preventing a specific virtual machine from occupying a resource of a PCIe fabric, when the resource of the PCIe fabric (in this case, a band) is shared and used by a plurality of virtual machines.
Further, in PTL 1, individual request queues are provided to a non-posted request, a posted request, and a completion, respectively. Further, by fixing a priority with which a request is extracted from these request queues, an ordering rule among requests defined in a PCIe specification is satisfied.
The ordering rule among requests in PCIe is a rule that determines whether or not passing is possible between requests, and between a request and a completion. For example, basically, a non-posted request cannot pass a posted request.
In most cases, this ordering rule determines whether or not passing is possible by a combination of a request classification including a completion, and a flag relating to an ordering included in a header of a PCIe packet or identification information on a request.
In PCIe, there is a configuration that determines two species of ordering rules, namely, a relaxed ordering (hereinafter, referred to as an RO), and an ID-based ordering (hereinafter, referred to as an IDO) by using information on a transaction layer packet (TLP) header of a PCIe packet.
Both of an RO and an IDO determine which ordering rule is applied by setting an appropriate value to an Attribute field of a TLP header. It is possible to use one of an RO and an IDO, or both of an RO and an IDO simultaneously.
By using an RO, a certain posted request is possible to pass a posted request that is issued prior to the certain posted request, a certain non-posted request is possible to pass a posted request that is issued prior to the certain non-posted request, and a certain completion is possible to pass a posted request that is issued prior to the certain completion.
Further, by using an IDO, passing between posted requests having a same request identifier, and the like are enabled. This request identifier is constituted of a combination of information on a tag field, and a requester identification (ID), which are included in a TLP header. The requester ID is an identifier in a PCIe fabric of a PCIe device that issues a request. A requester ID is constituted of a combination of three values, namely, a bus number, a device number, and a function number, and is also referred to as a BDF (registered trademark) number by using initials of the names of the respective numbers.
Further, PTL 2 describes that, regarding passing between transactions according to a PCIe specification, a posted transaction has a high priority, and is possible to pass a non-posted transaction or a completion.
Further, PTL 3 describes that, in order to efficiently handle multitudes of transactions that co-exist in a host bridge in both directions, namely, in an inbound direction and an outbound direction, inbound and outbound transaction requests are handled successively or non-successively, based on a specific classification of transaction and a defined ordering rule under control of a predetermined state machine.